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ABSTRACT 

This paper evaluated a 2-year^ 60 credit program 
which prepared educators for research positions within the public 
schools* Directed by the New York State Department of Education, ten 
New York state universities participated in the program* Following a 
brief introduction, the program design and recruitment procedures 
were discussed* Emphasis was placed on the areas of statistics, 
psychometrics, design, reporting, interpreting and constructive use 
of findings* The program design was: (a) course work in research 
methodology, statistics, measurement, and psychology; (b) research 
demonstration practicum; (c) field experience; (d) a 1-year public 
school internship* The results and effectiveness of the program were 
discussed, stressing the positive reaction of university directors 
and participants* Some conclusions drawn from the evaluation were: 
(a) the methods used in the program were successful; (b) graduates 
were able to use their skills in public school environments and other 
educational situations; (c) placement of graduates was possible 
without the aid of a special state certification* Recommendations 
included: (a) improvement of statewide seminars; (b) more emphasis on 
field work with possible orientation of certain program phases toward 
doctoral work* Nine tables and five figures of data were presented in 
the text* A U3-page appendix contained required courses, summaries of 
internship experiences, current employment of graduates, a graduate 
questionnaire on attitudes and employment, and professional 
achievement of graduates* (BRB) 
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FOREWORD 

The local school, according to the creators of the program herein 
described, is "where research is applied/* In order to determine if a 
particular schol^s program in or out of the classroom is effective or 
if a particular individual is, indeed, learning, proper research methods 
must be used and true evaluation must take place. 

The final report of the Program tells the story of the first Research 
Training Program to be cooperatively sponsored and operated by the New 
York State Education Department and various Nfew York State institutions 
of higher learning. The pioneering program, outlined in detail in the 
following pages, was initiated under Public Law 83-531, Section 2(b), as 
amended by Public Law 89-10, Title IV. 

Louis T. Di Lorenzo of the Education Department, the program's director, 
was assisted by two associate directors - William McLoughlin in 1967-68 
and Thomas Gould in 1968-69, Leo D, Doherty, Chief of the Bureau of 
Urban and Community Programs Evaluation, assumed program direction from 
1970 to 1972. 

Program directors in cooperating universities were: 

Paul Cullinan, New York University; David Fox, City College, The City 
University of New York; Elizabeth Hagen and Marvin Sontag, Teachers College of 
Columbia University; Esin Kaya, Hofstra University; Donald Meyer, Syracuse 
University; William McLoughlin, St. John's University; James Mitchell, 
University of Rochester; Donald Nasca, State University College at 
Brockpcrt; Reuben Rusch and John Rosenbach, State University of Nfew York 
at Albany; and John Skalski, Fordham University. 



Alan G. Robertson, Director of the Division of Evaluation and 
staff members of the Office of .Research and E aluation were involved in 
planning during the course of the program. Members of the Norcheastern 
Educational Research Association participated in meetings related to the 
program. Richard Borell, Joseph Foreman, Mary Horan, and David MacNulty 
worked on organizing and assembling material for the report. 

Preparation and writing of the final report was administered by 
John H. Rosenbach, assisted by Robert B. ladeluca and Loran Twyford. 
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I, INTRODUCTION 



In 1966, a premise was drawn by ttie New York State Education 
Department that to help improve public school instruction through 
research, it must be conducted in the schools themselves by local 
personnel with special research training and a practical orientation. 
At the time, positions for such personnel existed formally only in the 
large urban districts; in many other districts, the majority of research 
workers were not trained, but "drifted" into their positions. 

The premise above was based on prior research showing that in only 
a minimum of cases were university laboratory findings transferred to the 

public school setting. 

A Program for Training Educational Research Personnel for School 
Service was set up effective July 1, 1966, conducted cooperatively by the 
Department and selected New York State higher education institutions, and 
financed by 5 grants which totaled $1,432,284. From these grants $1,348,104 
was expended.^ 

The overall goal: To prepare educators to fiU research positions 

within the public schools. 

Specific duties of the public school researchers were listed as follows: 

1. Examine the continuing educational process so that problems 
hindering the reaching of objectives might be located and identified. 

2. Review research findings which might lead to solutions. 
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encumbrances. 



-l- 



3. Suggest possible solutions to these local problems. 

4. Examine current and new educational programs through the 
use of experiments and research studies. 

5. Develop evaluative measures of educational objectives. 

6. Evaluate teaching/learning materials. 

1. Use testing by: conducting surveys, instructing teachers 
in administering and scoring, analyzing and interpreting 
results, and preparing reports* 

8. Work with curriculum and evaluation committees. 

9. Conduct research inservice training. 

10. Cooperate with all interested and related agencies. 

11. Collect, record, and analyze required educational statistics. 
The means for arriving at the overall goal: Providing the educator 

with competency in areas of statistics, psychometrics, design, reporting, 
interpreting, and constructive use of the findings. 
The program, to achieve this, included: 

a. Course work in research methodology, statistics, 
measurement, and psychology. 

b. A research demonstration practicura. 

c. Fiela experience. 

d. A l-year public school internshxp. 

Three years of teaching experience were required of each candidate, 
enabling him to better put into practice knowledge gained in the program. 
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-11, PROGRAM DESIGN AND RECRUITMENT PROCEDURES 

A. Duties of State and University D i rectors 

On the State level , the director was responsible for: 

The establishment and maintenance of communication among units 
of the State Education Department, public schools, universities, 
and agencies on all levels. 
Consultation and advisory service. 
Review and approval of program reports « 
Dissemination of these reports. 

Conducting regional seminars on problems of educational 
research. 

Recruitment supervision and screening of candidates. 
Arrangement of field experiences and Internships. 
Conducting meetings. 

Department representation at related conferences. 
On the university level , the director was responsible for: 
Screening and acceptance of applicants. 
Transmission of required forms. 
Student advisement <> 
Supervision of field experiences. 
Approval and monitoring of internships* 

Coordination of the student's formal, university- based training 
with his practical off-campus training. 
B. Recruitment 

Descriptive materials, including posters and brochures, were distributed 
to elementary and secondary school principals, as well as to college and 
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university facilities. Administrators on ali levels were asked to recommend 
individuals previously indicating an interest in school research. 

C. .Selection Criteria 

Trainees were cooperatively selected by the State Education Department 
and the universities, using the following criteria: 

1. Completion of at least 3 years of elementary or secondary 
level teaching experience. 

2. Completion of no more than 12 hours in the required courses 
of the program. 

3. Graduate study admissions criteria of the particular institu- 
tion. 

Starting in 1968, trainees were required, in addition to submitting a 
written application (see Appendix A), to be interviewed by the State director. 
Such screening helped fo determine the candidate's understanding of the 
program rationale and his intention of working in an elementary or 
secondary school setting upon program completion. 

D. Structure and Organization 

The 2-year, 60-hour program consisted of three major components 
(model outline in table 1, institutional outlines in appendix B) . 

Table 1 

MODEL RESEARCH TRAINING PROGRAM 
Field Credits Required Courses and Field Experience Credits 

Principles, Methods, 9-15 Methods and Principles of Educational 3 

and Materials of Research 

Educational Research Educational Research Problems 

Research Dissemination Practicum 3 



Field 



Statistics 



Compute cs 



Table 1 (Cont'd.) 
Credits Required Courses and Field Experience Credits 
9-15 



E'^ucational and Psycho- 
logical Measurement 



Psychology 

Research in Substantive 
Areas 



3 

9-15 

3-6 
3-12 



Descriptive Statistics 
Statistical Inference 
Experimental Design 

Electronic Data Processing 

Principles and Theory of Measurement 
Test Construction 
Diagnostic Testing 



Psychology of Learning 

Student Choice-Research in Curriculum, 
Administration, Psychology, Guidance, 
Special Education, Sociology, and/ or 
Economics 

Field Work and Internship 12-14 Assistantship-First Year 

Skill development. One tull day per 
week or equivalent to be spent in 
appropriate field placement. Minimum- 
30 days in academic year. 



Credits 



Required Course Work 36 
Elect ives 10-12 
Field Experience 12-14 
Total 60 



3 
3 
3 

3 

3 



2-3 



2-3 



Summer Field Work 
Further skill development or early 
assignment to internship. Mriximtun- 
30 full days. 



Internship-Second Year 8 
Supervised experience in school research 
3 full days per week throughout 
school year or a total of 120 days. 



1. Course Work 

First academic year: In addition to those courses listed, a seminar 
on Educational Research Problems was offered by the State director at one 
upstate location and in the New York City area. Providing cohesion among 
the several university programs, it brought the trainees together on a 
regional basis to examine contemporary school problems, research studies 
underway, and the unique characteristics of school research. Regional 
personnel of national prominence were invited to address these seminars. 



Second academic; year: Courses were offered in research dissemination, 
and in research in substantive areas; e.g. curriculum, administration, 
or special education. The latter, along with the 10-12 elective credit 
offering, was to provide flexibility wichin ,x ' ^ and to further 

development of individual interests and specialities* 

2. Field Experience 

During the first year, from 30 to 45 days (see table 2) were spent 
developing basic research skills, with the assignment changing during 
the year according to needs. 

Table 2 

Suggested First Year Field Experience Activities 

Supervised Activity Minimum Days 

1. Developing a test, constructing behavioral 5 
statements of educational objectives, 

writing and reviewing test items, running 
an item analysis. 

2. Administering group tests and scoring a 3 
sample of tests* 

3. Observing the administration of individual- 3 
ized tests. 

4. Developing forms for the collection jf data 4 
and conducting interviews. 

5. Processing data manually and setting up 3 
data for electronic data processing* 

6» Observing the use and operation of a number 3 
of automatic data processing machines. 

7. Preparing tables, charts, slides, and other 5 
audiovisuals for the reporting of data and 
findings. 

8. Performing statistical computations. 4 
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3, Internship 

This took place the second >ear, emphasizing in-depth application of 
skills as an extension of the field experience. Its purpose was to develop 
interrelationships of these skills, with the intern spending three days 
a week throughout the school year (a total of 120 days) on a few major 
school-based studies. The trainee was encouraged to assume increasing 
responsibility* 

An advisory committee composed of university, public school, and 
cooperating agency representatives reviewed possible internship assignments. 
Selection criteria included: 

a. Nature and scope of study* 

b. Research activities planned for intern year* 

c. Qualifications of project director* 

d. Amount of time project director directed to study, plus 
his general availability for supervision. 

During the first 2 years of the program, the committee reviev;ed ongoing 
studies to locate those with maximum potential for interns in the following 
years* Locations were difficult to secure at the outset. As the program 
became known through questionnaires sent to local districts, boards of 
coopera'uive educational services, and educational and private institutions, 
requests for interns were received from project directors. Depending on 
geographical locations and number of trainees, possibilities increased for 
offering interns their choice of positions. 

Some internships became self -producing. A student would be placed in 
an internship and, upon program completion, become a staff member of the 
agency. At times, he would become supervisor of an intern placed in his 
former position (apoendix F) . 
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In most cases, intern supervision was shared between the project 
director and a university faculty member, with the director having a day- 
to-day relationship. It was suggested that the faculty member meet with 
the director and intern at least biweekly to evaluate progress, strengths, 
and weaknesses, and to plan activities. These visitations actually 
happened approximately bimonthly. In some cases, where the research 
project director was a university faculty member, the intern had only the 
one supervisor. 
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III. IMPLEMENTATION OF THE PROGRAM 

A. State Level Administration 

Louis T. Di Lorenzo of the Office of Research and Evaluation, New 
York State Education Department directed the program on the state level. 
Professional and clerical personnel from this office made constant contri- 
butions . 

William McLoughlin was appointed associate director during the program's 
second year. His duties included (1) assisting second-year trainees in 
finding on-the-job experience opportunities and supervision, and (2) 
assisting in establishing a new cycle of first-year trainees. Thomas 
Gould, a graduate of the program at Teachers College, succeeded McLoughlin 
in July 1968 for a 1-year period. In 1970 Mary Horan, a full-time 
research consultant, was hired. 

Regional seminars, carried by the trainees as a course in Education 
Research Problems, were given semimonthly at four different locations 
by the director and/or the associate director. In the New York City area, 
meeting places were rotated among the participating universities. Upstate, 
the seminars met at Rochester or Syracuse (1966-67), Syracuse (1967-68), 
State University College at Brockport (1968-71), and State University at 
Albany. 

Each seminar lasted 4 hours. Staff lecturers alternated with 
guest speakers, the latter including Robert Havighurst, University of 
Chicago; Ellis Page, University of Connecticut; Robel Ebel, Michigan 
State University; David Ausubel, Ontario Institute for Studies in Education; 



Philip Phenix, Teachers College, Columbia University; and Frederick Davis, 
University of Pennsylvania • 

Also John Flanagan, American Institute for Research; Donald Bitzer, 
University of Illinois; Clarence Spain, Schenectady (N.Y.) Public Schools; 
Daniel Stuffelbeam, Ohio State University; and John Stiglmeier, New York 
State Education Department, Speakers were available at all sites. All 
program participants were invited. Local educational researchers and 
recent program graduates occasionally participated in the seminars. 

A selected group of educational research specialists representing 
universities, school districts, boards of cooperative educational services, 
and various divisions of the State Education Department had, in previous 
years, made preparations for establishing a State Certificate for Specialists 
in Educational Research. Noting that the field of educational research 
parallels that of guidance counseling and school psychology, the group 
suggested that a similar certificate be created. It was assumed that 
establishment of state certification requirements vould lead to the develop- 
ment of training programs at various higher education institutions. 

In the early sixties, this group drafted suggested certification 
requirements. They were reviewed by research personnel at the 1964 
AERA convention and accepted with a generally positive reaction. They 
were also submitted to Ewald B. Nyquist, then New York State Deputy Commissioner 
of Education (correspondence from L. Di Lorenzo, 7/6/64), and other Depart- 
ment personnel. 

No action was taken by the Department* Certification procedures 
were being revised at that time and action on research certification was 
delayed. Di Lorenzo, who was involved in drafting the proposed require- 
ments, saw in Title IV legislation an opportunity to translate tha goals 
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of the draft into an education program supportable by Federal funds. The 
result was the Research Training Program, funded in July 1966. 

In the original proposal, it was stated that: 
...the State Education Department will be encouraging local 
school systems to establish new positions for Directors of 
Research for which those completing the training program 
would qualify. This will be done partly through establishment 
of the certificate for specialist in education (CRP #6-2705, p. 8). 
Until 1970, brochures describing the program included the statement 
"Candidates completing the program will be eligible for the New York 
State Certificate of Director of Educational Research which is to become 
effective in September 1969." 

Correspondence verifies the extended exfort to establish unique certi- 
fication which failed to evolve. School researchers in New York State 
fell into two categories already covered by existing regulations (Article 
XV of the Regulations of the Commissioner of Education). The first category 
authorized positions for educational researchers in supplementary services. 
The only certificate required was a valid teaching certificate (section 
1496). If more than 25 percent of the researcher's time entailed administrative 
duties, the second category, that of school administration or supervision, 
would apply, authorizing an administrative certificate under section 
119—' one which could be entitled director of research. 

Thus, in the view of the Bureau of Teacher Certification, there was 
no need to establish unique certification requirements for educational 
researchers (William Boyc' , Chief, Bureau of Teacher Certification, in 
correspondence with John Rosenbach, SUNY-Albany, 6/13/70). 
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For some trainees, the State's not establishing a unique certificate 
for research was a severe disappointment; but given the limited number of 
schools employing researchers, lack of such certificates is of questionable 
significance, especially since the position can be recognized another way, 

B. University Level Participation 

Ten universities, at various periods, participated in the Research 
Training Program. The periods of operation for each university are shown 
in figure 1 along with the number of trainees involved. Cooperating 
institutions were The University of Rochester; New York University; Syracuse 
University; Teachers College, Columbia University; Fordham University; 
St. John's University; State University of New York at Albany; Hofstra 
University; City College, The Citv University of New York; and State University 
College at Brockport. 

The first class of trainees began studies at seven universities in 
September 1966. Three of the ten original universities, although they 
had representatives helping to plan the project, decided not to participate. 
They were Cornell University, Queens College of the City University of 
New York, and State University of Ngw York at Buffalo. Replacing them in 
September 1967 was Hofstra University and, in September 1969, State Univer- 
sity College at Brockport and City College of The City University of New York. 

In appendix B are presented the educational programs of each of the 
universities. The names of the participating students are listed in 
appendix H. The initial program grant ran from July 1966 through August 
1969. Two continuation grants were received; (1) September 1969 through 
August 1970 and (2) September 1970 through August 1971. The latter was 
extended through June 1972. 
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Several universities withdrew from the program prior to its completion, 
Rochester admitted no new trainees after 1966, and was no longer involved 
after June 1968. The University of Rochester said that the program cost 
the university $3,950 per trainee per year, yet it received only $2,000. 
In 1969, two more institutions withdrew - New York University and Syracuse. 
Neither had admitted new applicants after September 1968. An officer of 
New York University cited the comparatively high cost of the program for 
such a small number of participantfi , stating that ''without a minimum of 
10 to 12 students, it is nearly impossible to provide even a minimum 
of quality standards." No reasoa for Syracuse University's withdrawal 
was given. 

In September 1969, Teachers College, Columbia University, admit ced 
only one student who, later in the year, left the program. Teachers College 
had no program during 1970-71, the last year of funding. Although no 
official reason was given for the piograir s phase outran official referred 
to the problems of recruiting high-quality candidates and the necessary 
"intimate" supervision of internships. 

The reasons given for the withdrawal of three ooi the original univer- 
sities are also of interest, especially in relation to the responses of 
university directors from the participating institutions (see Section IV, C, 
below) . 

Queens College, The City University of New York stated that the main 
stumbling block to CUNY's participation was the "assignment of 12-14 
credits for field experience." A shortage of applicants also contributed 
to its withdrawal. 
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Cornell University described the situation regarding trainee appli- 
cants to Cornell in the sununer of 1966: 

So far, we have had 45 inquiries, of v^hich only three are 
following up. The facts that the training program is so 
rigid and that trainees must also meet degree requirements 
(at Cornell) •••may be the reason for the low follow-up 
rate. None of the three who have expressed interest (the 
second time around) has completed the necessary application 
materials to be acted upon by the Cornell Graduate School. 
Apparently no applicants met all the state's or university's requirements, 
as no trainee entered the program at Cornell. Interest waned and no further 
effort was made to attract candidates. 

The only university to withdraw from the program even before effort 
was made to attract applicants was SUNY at Buffalo. An official cri- 
ticized several program details, and his comments regarding overall 
objectives are particularly relevant to the current (1971) employment 
opportunities of program graduates: 

I am not sure that the idea of placing a rather well- 
trained technologist in the methods of research is 
ideally the best way to attack the problem of research 
in the schools. I am sure that in the long run a 
better approach would be to persuade the people 
(college instructors) in the various professional areas, 
e.g. curriculum, student personnel, to produce research- 
oriented and at least somewhat research-methods-competent 
people (e.g. curriculum supervisors, guidance counselors). 
At the time when these people (e.g. newly trained curriculum 
supervisors, guidance counselors) were in the school. 
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1 believe that the availability of a high-level technician 
could make a great contribution by facilitating their work. 
On the other hand, placing him there now seems a litcle 
risky, as he may just blunder around and get everybody mad 
at research. 

In summary, institutions active at the end of funding (8/71) were 
three of the original seven (Fordham, St. John's, and SUNY -Albany) , and 
three which entered later (Hofstra, 9/67; City College, 9/68; and SUC- 
Brockport, 9/68), with Brockport continuing the program in 1971-72. 

C. Trainee Participation 

In the original proposal, the projected number of new trainees for 
the September, 1966 - June 1969 period, was 50 per year (CRP No* 6-2705, 
p. 8), or a total of 100 active students after the first year. In 1969 
(Continuation of Contract No* 0EG-0-8062705-3638(010) , June 1, 1969) and 
in 1970 (same as previous, September 1, 1970), only a total of 50 trainee- 
ships in any one year were requested. In table 3, a summary of the proposed 
number of traineeships for each year and the actual number of participants 
is given. The attrition of a typical class through the second year to 
graduation is indicated by arrows. The ratio between the proposed figures 
and those funded is approximately 2:1 for all periods except 1969-70. 
An explanation of this discrepancy follows* 

The U. S. Office of Education approved the initial proposal in 
June 1966, approximately 2 months before the first group of students 
were to begin studies. The participating universities, therefore, found 
recruiting candidates difficult. Three of the original ten universities 
withdrew, in part, because of failure to attract candidates. Thus, 
only 26 students were enrolled. 
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Table 3 

Proposed and Actual Number of Traineeships Per Year 





1966-67 




1967-68 




1968-69 




1969-70 




1970-71 


Prop. 


Act. 


Prop. 


Act. 


Prop. 


Act. 


Prop. 


Act. 


Prop. 


Act. 


First Year 


50 


26 , 




50 


31 , 




40 


28 




20 


24^ 




25 


6 


Second Year 








50^ 


■i^ 22 
1 




50" 


"1^19 




30 


25 




25" 


tl5 


Total Enrollment 


50 


26 




100 


1 




100 


'47 




50 


49 




50 


' 21 
1 


No. of Graduates 








50 






50 


1 

>H9 




30 


25 




25 




Percent Graduating 










77% 






61% 






89% 






54% 



An attempt was made to increase the number through further enrollment 
in January 1967, but the U^S.O.E. provided funds for a total of only 50 
trainees in subsequent years. A total of 53 trainees (31 first-year and 
22 second-year) participated, therefore, in 1967-68 • Total enrollment 
in 1968 was 47; in 1969-70, it was 49. A request for contract continuation 
was written in 1969, but only 20 new traineeships and 30 second-year ttainee 
ships were proposed. The U.S.O.E. granted one-half of this request and 
approved 21 new positions. Furthermore, five of eight trainees who left 
the program during the 1969-71 cycle were asked to do so after two or more 
semesters of participation. 

Following the arrows in table 3 will show the progress of trainees 
through the program. For example, of 26 enrolled in 1966, 22 entered the 
second year, and 20 graduated in 1968. Continuation into the second year 
by a Jtrainee indicated probable program completion. 



D, Expenditures 

Stipends and dependents allowances helped to support IC. trainees 
for up to 2 years each between September 1966 and June 1971, In addition, 
participating universities received funds to defray cost of tuition, equip- 
ment, instructors* salaries, and administration. The institutional allowance 

was a fixed sura per trainee per year, based on the university's designation as 
a private or state-supported institution. 

In table 4 is shown the amount paid to each university for allowance 
and student subsistence. The average contractual cost per student for the 
2-year training period was $9,870; the cost per graduate (77) was approxi- 
mately $13,600. 

The total of the grants for the Research Training Program by the 
United States Office of Education for the 5-year period starting in July 1966 
was $1,432,284 of which $1,348,104 was expended. Of the total expended, as 
of August 11, 1972, $1,162,036 was used for contract and institutional support.^ 
The remainder, approximately $186, 068, was used for State Education Department 
administrative staff salaries, evaluation, report writing and printing, supplies, 
materials, and travel expenses. Outstanding encumbrances at the time of writing 
this summary report are in the amount of $2,825.90 and are included in the 
above approximated total expenditure of $1,348,104 and in the State adminis- 
tration amounts. 

The unspent unencumbered balance of the last of the five grant awards, 
extended to June 30, 1972, amounts to $6,430,28. The total unspent unencumbered 

amount from the five awards over the 6- year extended contract period is 

o 

approximately $84,000, 



includes blanket contractual charges posted since 
the listing of institutional expenditures shown in table 4, p. 19, 

2 

Thus the $84,000 includes reawards from one contract to the next, 
^ -18- 
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lable 4 



Amounts Expended for Institutional Allowance 
and Student Support for Each University 
as of March 31. 1972 



Institutional 
Allowance 



Student 
Subsistence 



Total 



City College, CUNY $ 26,000.00 $ 51,327.00 

Fordham University 65,000.00 91,483.62 

Hcfstra University 49,958o33 82',100:i7 

New York University 22,242.00 36,070.00 

St. John's University 45,000,00 61,100.00 

sue at Brockport 48,800.00 84,670.00 

SUNY at Albany 59,000.00 90,100.00 

Syracuse University 18,000.00 28,475.05 

Teachers College, CoK Univ. 55,500.00 78,440,78 

University of Rochester 18,000.00 35,900,00 

Total $407,500,33 $639,666.62 



$ 77,327.00 
156,483,62 
132,058.50 

58,312.00 
106,100.00 
133,470.00 
149,100.00 

46,475.05 
133,940^78 

53,900,00 

$1,047,166.95 



1, 



1. 



Additional charges posted after March 31, 
See page 18, paragraph 3, for total. 



1972. 
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IV, PROGRAM RESULTS AND EFFECTIVENESS 

A. Total Enrollment 

Four classes completed a 2 -year cycle throughout the 5-ypar 
program with 107 formal enrollments and 77 (72%) graduations. Charac- 
teristics of graduates are compared in table 5 with those who failed to 
complete the program. In general, graduates tended to be somewhat younger 
more likely to be married, and to have more children. In addition, 79 
percent of males who initially enrolled were graduated, whereas only 56 
percent of the females completed the program. 

Table 5 

Demographic Data on All Trainees in the 
Research Training Program 
1966-1969 





Total 


Graduates 


Non-Graduates 




N » 107 


N = 77 


n = 30 


Mean Age 


33.5 


32.9 


35.2 


Mean Years Teaching 


7.2 


6.9 


8.1 


Percent Single 


17 


13 


27 


Dependent Children per 








Married Trainee 


1.9 


2.3 


1.5 


Percent Female 


30 


23 


47 


Percent Single Female 


13 


10 


27 



B. Trainees 1970-72 

Eight trainees, not included elsewhere in this report, entered the 
program in 1970. Summary data on these trainees are given in appendix E 

The Research Training Program was continued at SUC-Brockport 
under the direction of Louis T. Di Lorenzo who was on leave of 
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absence from the State Education Department. Partial financial support 
is being received through the State Education Department. 

Two trainees, aware of the imminent close of Federal funding, entered 
the program in 1970 at SUNY -Albany for 1 year. One is now completing the 
program through part-time study while the other transferred to the doctoral 
program in educational psychology. 

C. Reports of University Directors 

In November 1971, each of the 10 former university program directors 
was asked to assess the program now completed. Responses were received 
from 7 with reactions showing agreement in some areas and variations in 
others. Eight major areas for which assessments were asked and a sampling 
of reactions received are here listed: 

1. Admission of candidates^ including qualifications and methods of selection. 

Candidates had to meet two sets of admission criteria those at the 
State level and those of the respective universities. The State Education 
Department required 3 years of teaching experience. At the university level, 
the criteria were varied, including such items as academic record, Miller 
Analogy scores, and Graduate Record Examination scores. At both levels, 
recommendations and personal interviews were used. 

Regarding a possible difference between research training program 

participants and graduate students in doctoral programs, some of the 

comments received were: 

"Not many of the candidates who actually submitted applications... 
had outstanding academic records." 

"Same as for M.Ed, students." 
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"In general, the candidates selected turned out to be energetic, 

capable, and interested in research. ...the average aptitude 

score of those admitted to the program was somewhat below those 
of Ph.D. candidates," 

2. Purpose of the program . 

All respondents agreed with the original premise that the program's 
studies should be oriented toward public school needs with an emphasis 
on a research demonstration practicum. In only one university did the 
director report the students and instructional staff being unclear about 
the purpose of the program. The institution later withdrew. 

3. Suitability of academic courses . 

The majority of institutions felt their courses to be relevant to 
the program. 

"The program was administered within the Department of Educational 
Psychology, which is one of the largest in the northeast. . .it is 
our conviction that the coursework available to the trainees was 
highly appropriate." 

"The courses we provided were suitable." 

"A strong public administration program has been supplemented 
by newly developed technical courses in research and statistical 
analysis." 

"Suitability of courses: Excellent we chose them." 

"I believe the typical diet of courses in the program did not 
fully satisfy the needs of school-based researchers. I, 
personally , believe the types of research, measurement, and 
statistical skills we teach in college are designed for 

'classical research situations.' What is needed in school 
evaluation falls considerably short of this model. Courses 
more in keeping with what one finds in school evaluation are 
needed." 

4. Judgments of participants' achievements in the program . 

The majority of respondents were satisfied with the trainees' growth 
during the program. Specific comments ranged from "adequate** to "well 
above average in ability and maturity and in response to the program." 
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Some university representatives said that program effectiveness would 
be best judged in terms of subsequent employments 

5. Placement of participants in program- related employment . 

The majority of the program graduates, even those who returned to 
classroom teaching, are using their training in some aspects of their 
work (see table 6, below). Research positions in many schools were not 
available, and some of the trainees either returned to their former 
responsibilities or continued their graduate education. 

Three of the seven respondents described extreme difficulty in 
finding appropriate positions for trainees. Definition of success depended 
upon the interpretation of the term, "program-related employment." 

6. program's effect on curriculum or administrative change . 

Those higher education institutions with well-developed programs 
in educational research changed little. One developed a master's 
program in educational research. Others developed new courses to accommo- 
date the Research Training Program, these courses coinciding with a growth 
of specific departments at each university. 

7. Suggestions for improved program operation . 

Improvement ideas fall into three categories, with numerous suggestions 
being given as shown below. 
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a. Overall goals of the program : 

"The willingness of the cooperating institutions to accept 
the goals of the program must be clearly established." 

"The program might have benefited from better communication 
among the participating institutions.** 

b . Administrative problems : 

"From our point of view, the admission of candidates and 
their ultimate selection of the college of choice came 
too late in the academic year for most efficient selection. 
We were faced with the need to accept applicants simply 
to have a viable program continuing." 

"Probably the greatest administrative difficulty we encountered 
centered around the uncertainty of the number of traineeships 
available for each year and the lateness of applications. 
From our vantage point, it would have been helpful to have 
received applications in the early spring cind alao to have 
known more precisely how many trainees we could accept." 

"Confused and often conflicting selection procedures." 

c . Coursework : 

Some directors reported that the internship 
features of the program were poorly designed and that there 
was a conflict between part-time involvement in internship 
training and in academic education. One found the program "geared 
more to preparing technicians than research directors" and 
proceeded to develop a "doctoral program using the Research 
Training Program as a starting point." 

In contrast, the graduates' impressions of both regional 
seminars and internships were generally favorable, and many 
of them continued to work toward higher academic degrees. 
Several directors pointed to the fact that few school districts pro- 
vide either the time or money needed to employ school researchers. A 
number of them mentioned the lack of demand for graduates at the level 
described in the proposal. One suggested a "combined program, funding 



-24- 



the position (at the school level) and supporting the candidate" in the 
manner, perhaps, that guidance services and training were funded in the 
early sixties* Other suggestions included having students take courses 
in administration in preparing for administrative positions which have 
research and evaluation responsibilities, and encouraging program graduates 
to continue study toward a clearly defined terminal degree such as a 
doctorate . 

8. Should this or a similar program be continued? 

All of the respondents said that this should be done. 

9. Estimate of the professional qualifications of program graduates . 

All respondents were satisfied with the professional qualifications 
of the graduates. Comments ranged from "adequate after they were trained" 
to "some of the most qualified people in the area to do the types of 
research and evaluative jobs most schools have need doing." 

Summary 

University directors were in agreement as to the need for educational 
researchers in public schools and the need for an educational program to 
prepare them. Although careful to distinguish between the intellectual 
ability of these trainees and doctoral students, their reaction to the 
trainees' achievements went from "adequate" to "very acceptable," the 
few exceptions being unqualified participants who were counseled out of 
the program. Program graduates were unanimously judged to be professionally 
qualified* 

All directors felt t!ie program should be continued. Minor differing 
views were expressed on the proper implementation of the program goals. 



details of administration, the State-directed seminars, and field experiences, 
with the only major difficulty, as all directors saw it, being the lack of 
employment opportunities in this area of responsibility. 

An apparent difference existed between the climate at upstate insti*^ 
tutions and those in the New York City metropolitan area. This was not 
due solely to the geographical separation. With the upstate institutions 
still growing at a rapid pace, the Research Training Program became an 
integral part of this growth and helped to contribute to it. At one 
institution, the initial six trainees were a large proportion of the 
total post-master's level students in the Department of Educational Psychology. 
In another, no doctoral level programs existed, making the 60-hour training 
program one of its most advanced graduate programs. 

Trainees at both these institutions were considered among the academic 
elite and a strong esprit-de-^corps developed. Each of these institutions 
also had its own "regional seminar," possibly adding to the cohesiveness . 
Upstate university directors, furthermore, had far more direct contact with 
the trainees. In short, the program was profitable and relevant to all 
concerned. 

Large metropolitan areas have their problems in all walks of life, 
this particular training program being no exception. In the downstate 
area, student involvement with professors and fellow-students was not as 
intense, one possible cause being a larger number of interinstltutlon 
transferees. One university viewed trainees as comparable to M.Ed, students* 
Sheer enormity of student populations might also have affected the program. 
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D. Responses of graduates to questionnaire 

A questionnaire was sent to all 77 graduates asking (1) current 
academic and employment information and (2) their attitude toward the 
program. Concerning their employment, graduates were asked such salient 
facts as their present salary, salary before participating in the program, 
relevancy of their present responsibilities to their courses and intern- 
ship, and comparison of their prior employment expectations with their 
present position. 

They were asked whether or not they felt the program had adequately 
prepared them, the part of the program they considered the most valuable, 
and if they would make any program changes. Graduates were also asked 
to state the types of degrees, certificates, and other academic creden- 
tials they now held. There were 44 questionnaire replies as of 12/31/71.' 

Employment 

Of the 77 graduates, 37.6 percent are presently employed by local 
districts, including urban as well as Union Free School Districts (see 
figure 3). Responsibilities on a local district level are held primarily 
by those designated as researchers or evaluators, with the remainder being 
principals, supervisory personnel, guidance directors, and classroom 
teachers . 

Another 24.7 percent of the graduates are now allied with university 

research centers or instruct on a college level. An additional 18 percent 

are with supplementary private or public educational agencies, on either 

a state or local level. The proportion of time devoted to research and 

evaluation activities by type of employment is shown in figure 4. 

*A comparison of emplojrment of respondents and nonrespondents 

in figure 2. In this respect the two groups appear highly similar. 
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Figure 2. A Comparison by Employers of all RTP Graduates (Population) with 
Graduate Questionnaire Respondents (Sample) 
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AVERAGE RATINGS OF RELEVANCE OF OVERALL RESEARCH TRAINING 
PROGRAM TO CURRENT EMPLOYMENT 
(Scale 1 low to 5 high) 

Figure 5 
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A further statistical breakdown of the above is shown in figure 3 
along with the percentage of those unemployed, continuing their studies, 
or employed with educationally related private firms. 

Questionnaire answers relate the respondent's type of employment 
to his expressed attitude toward the program's relevance (see f igure 4). 
Four out of five who are employed by local school districts rate the 
relevance as ^'medium," whereas four out of five employed by supplemental 
agencies rate relevance **high." Even so, the vast majority of graduates, 
on an overal basis, rated the program as relevant to their current employ- 
ment (table 6) . 

Table 6 

Relevance of Research Training Program to Current Employment 

Relevance No. of Respondents 

High 24 

Medium 16 
Low 

Total 41 

Graduates were asked to indicate the extent of involveiuant with 
each of 11 duties described in the original proposal as functions 
of a public school research director (see table 6 for complete list). 

Two activities received a relatively large number of "Always" responses 
(1 and 5). Activity 1 states, "Examine the ongoing process of education 
in order to locate and identify problems impeding the achievement of educa- 
tional objectives." This activity is the most general and includes a 
variety of possible functions. Activity says, "Develop evaluative 
measures of educational objectives." This activity is more specific and 
was stressed in the program. 
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Three activities, (6, 9, and 11) received a relatively larger 
number of "Never" responses. They read: "Operate projects in which teaching 
and learning materials will be evaluated," "Conduct inservice training 
in conjunction with research projects and in the application of research 
findings," and "Collect, record and analyze educational statistics required 
by the board of education, the superintendent of schools, and municipal. 
State, and Federal agencies." 

Table 7 

Extent Graduate Respondents Perform n 
Duties of a School Researcher/Evaluator 

Activity Always Sometimes Never 

1 Examine educ. processes 

2 Review research 

3 Suggest solutions 

4 Evaluate programs 

5 Develop evaluative techniques 

6 Evaluate materials 

7 Tests 

8 Consultant-curriculum 

9 Inservice training 

10 Cooperate with other agencies 

11 Educational data 



Levels of Education Attained 

There were no doctorates among the 77 graduates upon their entry 
into the program. Of the 77, 73 percent had master's degrees and 27 percent 
only bachelor's degrees. Doctorates were later received by 20 percent, with 
32 percent earning degrees beyond that of the mASter^s (e. g.; Certificate 
of Advanced Study, University Certificate, Professional Diploma). 

Of the 35 graduates who do not have doctorates at this time, 12 
are continuing graduate study with a view toward earning a doc- 
torate in the near future. It is anticipated that 85 percent of the total 
eventually will obtain higher degrees* 



20 18 3 

6 31 4 

8 30 3 

9 24 8 
12 24 5 

6 22 13 

9 24 8 

8 30 3 

6 20 15 

8 26 7 

8 17 16 
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Professional Achievements 

Membership in professional organizations, books and articles written, 
conference presentations, and studies conducted are other indexes that 
may be related positively to the success of the program. 

Professional membership was reported as follows: American Educational 
Research Association (AERA) — 66 percent; Northeastern Educational Research 
Association (NERA), with one on the board of directors ~ 61 percent; Phi 
Delta Kappa 32 percent; National Council on Measurement in Education 
(NCME) ~- 15 percent; and American Psychological Association (APA) — 5 
percent. 

Membership was reported in 25 other organizations related to education, 
psychology, administration, and subject areas or in local school organizations 
Dapth of activity range from one respondent who is in six national organi- 
zations to a few respondents who specified "none." 

A list of 142 publications, presentations, and studies conducted by 
graduates of the Research Training Program is provided in a ppendix J. 
This averages 3.2 per graduate as of December 1971. 

Graduates^ Attitudes Toward Program 

The graduates were asked to evaluate or comment on several aspects 
of the program, including program administration, "most valuable experiences, 
course work, internship, employment opportunities, program deficits, and 
personal attitude changes. 

(1) Administration 

Administration was rated using a 5-point scale, on both a university 
and state level (table 8). University administration was judged "excellent" 
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by 52 percent of the graduates, with the remainder of the responses spaced 
over the other four rating categories. Seven percent rated it "low". At 
the State level, an "average" rating was the most frequent response, with 
35 percent so stating. The average rating of the university administration 
was 3.93 or slightly less than "good." The average rating of statewide 
administration was 3.56; between "average" and "good." 

Table 8 

Ratings of Program Administration 
(Scale - 1 low to 5 high) 





Student Rating 


Low 
1 


Less 
2 


Aver. 
3 


Good 
4 


Excel. 

5 


Total 


University Level 
Administration 


No. of Responses 


3 


4 


8 


5 


22 


42 


Percentage 


7.1 


9.5 


19 


11.9 


52.4 


100 


State Level 
Administration 


No. of Responses 


4 


2 


15 


10 


12 


43 


Percentage 


9.3 


4.7 


34.9 


23.3 


27.9 


100 



(2) Most Valuable Part of the Program 

The internship experience was cited by 30 percent of the graduates 
as the most valuable part of the program, with 17 percent mentioning 
academic coursework in general or specific classes such as research 
methodology, computer programing, or statistics courses. Others cited 
the value of regional seminars or the opportunity to interact with fellow 
trainees, competent instructors, and school personnel. Some responded in 
a very general way, praising the entire program or the exposure to research 
in educational psychology. 
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(3) Suggestions for Improvement 

A variety of suggestions were made to strengthen the program. Certain 
ones, especially those pertaining to the course nature and content, were 
more relevant to a particular institution then to the program as a whole. 
Other suggestions relate directly to current employment and may be 
considered with respect to a graduate's specific problems rather than to 
the basic objectives of the program. 

The most frequent suggestion called for increased emphasis on practical 
applications of research and evaluation skills or for more field work. 
Eleven persons specifically requested more emphasis on applied research. 
Seven students felt that the program should lead to an Ed.D. or a Ph.D., 
with more course work, if necessary. 

An additional suggestion was that provisions should be made, within 
the program, for job placement. Several respondents asked for an emphasis 
on data processing, others requested the inclusion of courses on multi- 
variate analysis, and three thought deletion of administration courses 
would improve the program. 
Other suggestions: 

More stringent internship entrance qualifications, 
increased emphasis on public school orientation, 
fewer project centers (increasing unity and communication), 
closer supervision by the director, and 
limitation of coursework participation to trainees. 
One graduate thought the State and college directors were not suffi- 
ciently committed to the program and students. Another graduate stated. 
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"The changes that have to be made are in the legislative area, so that 
meaningful research can be conducted, and not behavioral objective, dead- 
line-oriented research." 

A. Coursework, Internship, and Employment 

Graduates were asked to rate the quality and effectiveness of academic 
course work, internship, and postprogram employment opportunities. These 
were rated on a 5-point scale, 5 being the highest. 

Academic course work: The quality of "academic coursework" averaged 
4.02, ranging from 2 to 5, with 4 the most frequent rating. "Effective- 
ness of academic coursework" was rated 4.0 on the average, ranging from 3 
to 5, with 3 and 5 chosen 15 times each. 

Internship: This was given the highest average rating of the three 
areas, "quality" averaging 4.18 and "effectiveness" 4.19. 

Employment opportunities: The least favorable aspect of the program 
was "Post-Program Bnployment Opportunities." "Quality" received an average 
rating of only 2.83, with ratings of 3, 2, and 1 accounting for 29 of 42 
responses. "Effectiveness" was even lower in the opinion of the graduates, 
with an average rating of 2.79. Possibly relevant is the fact that seven 
graduates did not even bother to answer this part of the question. 

The rating of these three major program aspects are summarised in 
table 9. 

B. Program Deficits 

Graduates were asked if there were any current research or evaluation 
problems with which they were dealing and for which the program had not 
adequately prepared them. Slightly over 50 percent reported they had 
received adequate training and that there were no problems. One answered 
"OF COURSE. However the program gave enough background for mc to continue 
learning new things." Three individuals indicated that the question was 
not applicable to their present positions. 
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Table 9 



Graduate Ratings of Three Major Aspects of the Program 
(Scale « 1 low to 5 high) 



Aspect 


Number of Graduates Giving Rating of 


c 

J 


/, 


J 


0 
L 


1 


Total 


Average Rating 


Academic Course Work 
Quality 


13 


21 


8 


2 


0 


44 


4.02 


Effectiveness 


15 


12 


15 


0 


0 


42 


4.00 


Internship 
Quality 


22 


11 


9 


1 


1 


44 


4.18 


Effectiveness 


22 


11 


8 


0 


2 


43 


4.19 


Post-Program Employ- 
ment Opportunities 
Quality 


8 


5 


10 


10 


9 


42 


2.83 


Effectiveness 


7 


5 


7 


8 


10 


37 


2.76 



Of the remaining responses, seven stated that training in evaluation 
could have been more comprehensive, especially in the areas of "affective 
domain" and test construction. Five felt that their preparation was not 
adequate in research design and practical experience. Five indicated 
that more statistics, especially in multivariate analysis, would have 
been helpful. Four respondents had looked for more training in human 
relations and interpersonal dynamics. Three said they could have profited 
from more work with computers and data processing. 

Course-related criticisms appear, on examination, to be more appli- 
cable to particular institutions or individual situations than to the 
program as a whole. 
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C. Attitude Changes 

Graduates were asked if their atticudes toward the program had changed 
since completion of training. "No change" was reported by 86 percent. 
Many addsd, in fact, that they retained strong positive feelings about 
the program. Of the six who expressed a change in attitude, five were 
affected by the realities of their public education experiences differing 
from the theoretical approaches and attitudes developed during the program. 
One felt he found it necessary to change from a re'search orientation to 
an evaluation orientation. 

(4) Other Comments 

Each person was asked to list any reflections, observations, or 
comments beyond those previously stated ralating to any facet of the 
Research Training Program. Most frequently, graduates mentioned the 
extreme limitation of job opportunities in research and evaluation, 
especially on a public school level. Many thought the program should have 
led to a doctorate. Some thought that certain courses and/or the intern- 
ship should have been more extensive. Most respondents expressed very 
positive feelings toward the program and its potential implications for 
the field of education. 

A. Analysis of Internship 

Internships lasted an average of 122 days: 77 days were spent in a 

central office, 27 in the field, and 18 at other locations. Students 

earned an average of 8.6 credit hours for the internship. 

According to 51 percent of the trainees, the internship projects were 

organized^ yet flexlBle enough so that they, themselves, could suggest changes. 
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Another 41 percent stated that in their internships, few, if any, procedures 
were established; that the intern developed and applied procedures appropriate 
for the project* Thirty-three percent worked on six or more projects, 
22 percent on four or five projects, 20 percent on two or three projects, 
and 25 percent on only major projects. 

More than 50 percent said that they had worked the equivalent of two 
or more days with teachers, school administrators, other school staff, 
college faculty, other researchers, and with children* Little contact 
was made with governmental personnel, parents, boards of education, or 
community leaders* 

University directors visited internship locations an average of 
2.8 times per year. Interns, however, met with their university directors 
an average of 6.3 times. Fifty- four percent of the interns said that 
they conferred with their project director as often as necessary, with 
26 percent stating that this was almost daily. Forty-eight percent said 
they were responsible to other persons in addition to their director. 

No written report was required by 43 percent of the interns, 34 
percent said that frequent reports were required and, in the case of 23 
percent, only an end of internship report was required. The majority of 
interns found that the following facilities readily available: typing 
service, desk, telephone, work space, libraiy and reference materials, 
calculators, computer services and test files. 

The internship was an extremely valuable experience and one that 
all trainees should have, according to 67 percent of the respondents. 
Twenty-seven percent described their internship as valuable, but indicated 
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that others should be considered before trying the one they had. The 
remaining six percent said that their internships were of minimal value. 

Summary 

In general, current employment of the graduates is in areas consonant 
with the program objectives. Most of the sample of graduate respondents 
view their program experience positively. They view the overall program 
as relevant to their work. Although the time spent on various duties 
varies, most of the specific responsibilities of a Public School Research 
Director, as described in the proposal, are being performed by the majority 
of the graduates in their employment. 
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V. DISCUSSION 

Statewide Research Training Program 

In the program, 10 universities planned and administered coopera- 
tively a Research Training Program under the direction of the New York 
State Education Department, With Federal financial funds an important 
aid, the program was largely a success. 

Areas of difficulty seen in retrospect can be overcome in future 
programs. At some universities, a program composed of less than 10 
students is difficult, at times causing inferior results. In a univer- 
sity having a strong doctoral program, provision for transitioning research 
trainee graduates to such a program might be made. In some cases, a 
university should direct its energies to either one type of program or 
another. Certain universities might consider staying away from a 60- 
hour program unless it leads to a doctorate. 

Employment Opportunities for Graduates 

Fluctuations of the job market make it difficult to plan such a 
training program in relation to employment. This study does not compare 
graduate placement with training received. Taking specialized training 
appears to have the risks of not being appropriately placed, or of even 
being unemployed. It is crucial, therefore, that employment placement be 
considered in advance, and be a continuing concern by both trainees and 
those conducting the program. 

While one possible solution would be to plan training which is 
responsive to job demand, difficulties arise with the combination of a 
2 •year program and inflexible funding. Funding initial employment. 
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addition to the program itself, might be a beneficial investment. 

Specialist and Doctora l Program 

A comrnitment to either a specialist or doctoral program need not 
be an irreversible decision. A doctoral candidate may find that a 
specialist degree more adequately meets his needs, and vice versa* A 
program night also be developed flexible enough to accomodate the 
candidate's changing demands. 

Practical Field Experiences 

Several of the graduates strongly recommend that practical field 
experiences be provided beyond those included in the program. A proper 
balance would, of coutse, have to be struck between practical and theo- 
retical approaches. Individual differences among students, as well as 
universities, would need to be considered. Exposure of the student to 
the relative merits of each approach would help him to make a more 
enlightened decision concerning his own course of study* 
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VI. SUMMARY AND CONCLUSIONS 

Problems 

Many educators feel that the use of more pertinent research findings 
can strengthen the education of our nation's children. The purpose of 
the Research Training Program was to prepare researchers who, with a prac- 
tical orientation, would apply their findings and methods to classroom 
decisions and operations. 

When the program began in 1966, there was an urgent need to evaluate 
many of the programs assisted by the Elementary and Secondary Act, as well 
as other Federal and State categorically aided programs. There was also 
a severe shortage of qualified research personnel. 

Approach 

The U. S. Office of Education awarded a grant to the New York State 
Education Department to organize a 60-credit, 2-ye0r sequence for 
training public school-based researchers. Directed by the Education 
Department, the program was cooperatively conducted by 10 New York State 
institutions of higher education. The course work included research 
methodology, statistics, measurement, psychology, a research demonstration 
practicum, field experiences for the development of skills, and a L-year 
internship in a school-based study. 

The universities organized their own study programs and were respon- 
sible for counseling the trainees' use of time, advising of course work, 
and locating internships and supervised field experiences. It was assumed 
that the shortage of school research personnel would continue, and that 
the establishment of a State certificate would assist in job placement 
for the graduates. 
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Results 

The program enrolled 107 trainees; between September 1966, and June 
1971, 77 were graduated. The average cost per graduate to the program 
was $13,600. University directors judged the graduates to be professionally 
qualified and recommended that the program be continued. They distinguised , 
through, between trainees and other doctoral level graduates in terms of 
ability. 

The program was influential in facilitating the growth of research 
departments at two institutions. The anticipated special State certifi- 
cation for graduates has not materialized. 

School districts employed 38 percent of the graduates, with 25 percent 
going to universities and 18 percent to State and regional agencies. 
Graduates are finding their current empio3mient relevant, in varying degrees, 
to the education received in the program. The amount of time spent in 
research and evaluation activities ranged from 4 percent for secondary 
administrative personnel to almost 70 percent for research and evaluation 
positions at the secondary and collegiate levels. The mean amount of time 
spent on research and evaluation activities was 54 percent. 

After completing the program, 64 percent of the graduates continued 
on to receive higher degrees. An additional 20 percent are aiming at this 
goal. Within the 5 -year period of the program, each graduate prepared 
works for publication, conducted studies, or gave presentations at 
professional meetings an average of 3.2 times. 

Graduates rated university administration of the program as "good," 
with State-level administration rated between "average" and "good." The 
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second-year internship experience was cited by the graduates as the most 
valuable part of the program. On a 5-point srale, the internship t-'bs 
rated highest with 4,2, academic coursework - 4, and a 2,8 rating for 
postprograra employment opportunities • 

The most frequerit suggestion for improvement was for increased 
emphasis on practical application of research and evaluation skills or 
more field work. Many felt that the program should lead to Ed.D. or a 
Ph.D., with more coursework, if necessary. Student attitudes toward the 
program were found to be positive and unchanging. Graduates, nevertheless, 
were disappointed that more research and evaluation jobs were not avail- 
able in a period when employment of educators in general was low. 

Cone lusions 

1. Methods used to prepare research and evaluation specialists 
through a coordinated program proved successful. 

2. Graduates were able to use their specialized education effectively, 
not only in public school environments, but in other educational 
situations. 

3. Although placement of graduates was not ideal, it was found to 
be possible in a period of low employment, and without the aid 
of a special State certification. 

4. In future programs, the organization of statewide seminars could 
be improved. 

5. More emphasis should be placed on field work, with a possible 
orientation of some of the program phases toward doctoral work. 
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VII . RECOMMENDATIONS 



In reaction to this study, a number of followup activities, on both 
a statewide and nationwide basis, are recommended « 

1. Employment of graduates 

Not all of the graduates are in positions making greatest use of their 
preparation and experience. A list of graduates available for placement should 
be maintained and distributed where action is most likely to be taken. Names 
should be accompanied by a detailed description of the researcher's experience 
and education. 

2. Certification 

Certification in the United States is undergoing radical changes. The 
specific certification of yesterday is giving way to the more general certifi- 
cation of today. Reciprocal certification among states is becoming the rule. 

Performance criteria are being more regularly used as a basis for certifi- 
cation. There is an emerging trend for professional organizations to assist 
the Department in establishing certification standards in cheir areas. A 
possible use of this latter approach should be considered in the certification 
of research personnel. 

3. Program continuation 

The need for advanced preparation of research and evaluation personnel 
to serve on a public school level continues to be urgent. Those universities 
and colleges that have developed strong programs, as a development from this 
federally-funded Research Training Program, should be encouraged to continue 
and refine them. As demands for quality education and accountability 
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Increase, with possibly continued Federal financial accistance for local 
operation, the need for such personnel will expand even more. 

4. Program Information Dis s emination 

Local school administrators in a position to employ graduates of 
such programs as described herein, may not be aware of their availability. 

5o Teacher Applications 

Modern teachers need training in applying research to pupil learning 
situations. A cadre of local personnel having research training should be 
developed to train teachers to locate proven practices, view demonstrations, 
and adapt materials and processes to their own planningc 
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Appendix A 
Office of Reiearch aad Evaluation 
York State Education Department 
GRADUATE TRAINING PROGRAM FOR SPECIALISTS IN EDUCATIONAL RESEARCH 

Application Blank for Graduate Training Program 

IDENTIFICATION 



Name; 



Address;* 



Unlverilty:_ 
Soc. Sec. # 



Tchrs. Ret. 

Blrthdatei 

Phone: 



DEPENDENTS 



Name 



Relationship 



A££ 



Name 



Relationship 



Institution 


Decree 


# Credits 


Major Field 


Dates Attended 







































TEACHING AND OTKEl 


I SCHOOL EXPERIENCE 


Position 


Subjects, Grades, or Areas 
Taught, Supervised, or Admin: 
Istered 


School and Location 


Dal 

From 


ties 

To 

































area listed,) 



Tests and 

Measurements 
Psychology of 

Learning 
Educational 

Psychology 



Under 
Grad« 



Grad. 



Statistical 
Inference 

Advanced 
Statistics 

Principles, 
Problems , 
and Methods 
of Research 



Under 
Grad. 



Grad. 



Under 
Grad, 



Grad. 



Research 
Design 

Test 

Construc- 
tion 

Computers 

Electronic 
Data Process. 



*Plea8e nr-.lfy the Director of Research Training, Office of Research and Evaluation, New York 
State Education Department, Albany, New York 12224, of any change In mailing address. 
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Appendix C 

Survey Form of Potential Intern Locations 

THE UNIVERST'^Y OF THE STATE OF NEW YORK 
The State Education Department 
Research Training Program 

Survey for Research Training Internships 



Respondent: 
Address : 



Zip Code 



Title of Research Underway or Contemplated (If you have two or more projects, 
attach additional sheets answering items 1 through 10 for the second and other 
projects,) 



2. Duration of Study 



3. Nature of Study (Describe in brief the area of Investigation and methodological 
approach. Attach proposals and reports or project summaries If available.) 



4. Personnel Engaged in Project (If part-time, indicate hours per week spent on study •) 

M rr'.i Full-time or 
Name Title , ^ ^. 
part-time 



-53- 



5. Source of Project Support (e.g,; USOE, University or foundation grant, school 
budget allocation, State aid) 



6. Would you be able to use the services of a second-year graduate student on your 

project 3 full days a week for the 1967-68 school year? if so, how 

many students? " 

7. What would be the nature of assignments for the intern or interns? 



8. Who would be responsible for supervision of the interns? 



9. Indicate the Training and Experience of the person designated above. 

College Decree Date of degree 



Research Experience 



Publications 



10. Would you be in a position to offer any compensation to an intern? 



l/U/67 



Appendix D 

List of Potential Work Opportunities While Interns 

Mr, Robert M. Hecht Various projects dealing with college 

Bronx Community College student activities and achievement. 

School Budget 



Martin B. Miller 
Yeshiva University 
U.S.O.E. 

Leonard Diller 
N.Y.U. Medical Center 
Institution Grant 



Mr. Robert N. King 
Glens Falls City Schools 
A - Title III, E.S.E.A. 
B - Title I, E.S.E.A. 



Elizabeth M, Koppitz, 
BOCES - Meadow Brook 
BOCES 



Millie Almy 

Teachers College, Columbia Univ. 
U.S.O.E. - Teachers College Faculty 
Research Fund 

Josefa Nina Lieberman 
Brooklyn College 
NIMK, U.S.O.E. 

Mr. Arthur I. Gates 
Institute of Language Arts, Teachers 
College, Columbia University 

Mr. Nathan S. Washton 
Queens College 
None 

Mr. Julian Roberts 
Yeshiva University 
State Experimental Program 

Mr. Benjamin Cohn 

BOCES - Westchester (Yorktown 

Heights) 
U.S.O.E., N.Y.S. Educ. Dept. 



Curiosity behavior in educable 
mentally retarded adolescents: 
characteristics, modif iability , and 
training. 

Develop schemes of observation of 
behavior of brain- damaged children 
in the nursery school and primary 
grades . 

A - Improving education for inter- 
national understanding. 

B - The effectiveness of remedial 
techniques used for working with 
individual pupils. 

Children with emotional and/or 
neurological problems. Also^ con- 
struction of a memory scale for 
elementary school children. 

Effects of logical thinking in the 
second grade. 



Playfulness as a clue to cognitive 
styles in adolescents. 

Statistical studies of methods of 
teaching reading and testing reading 
abilities. Studies cf the general 
theories involved in teaching reading. 

Taxonomy of pupil questions in 
science for creativity. 

Human relations and its effect on 
teaching- learning in Social Studies. 

Four studies - 1) Identification of 
learning in first and second grades, 
2) Test construction, 3) Data pro- 
cessing, and 4) Counseling under- 
achievers. 
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Mr. Daniel Ringelheim 

Washington Square 
NICHD or U.S.O.E. 

Mr. Joel Elkind 

Ramapo Central School District #2 
State Aid, E.S.E.A., School Budget 



Philip A. Bolger 

New York City School System 

CRP, E.S.E.A. 

Mr. Herbert Rusalem 
Hunter College 

Vocational Rehabilitation Adminis- 
tration 

Brother Aloysius Rafael, F.S.C. 
Bishop Loughlin Memorial High 

School 
School Budget 

Howard F. Fehr 

Teachers College, Columbia Univ. 
U.S.O.E., Teachers College 

Mr. Leonard W. Ingraham 

Board of Education, Instruction and 

Curriculum, New York City 
School Budget, E.S.E.A., State Aid 

Jack Bernard 

North Belmore School District 
School Budget 

Mr. Josef E. Gorai 

Pratt Institute 

School Budget, Private Funds 

Mr. Irving Zweibelson 

New Rochelle Public Schools 

School Budget, State Aid 

Mr. Roger Reger 
Williamsville School District 
Local Funds 

John M. Dodd 

State Univ. of N.Y. at Buffalo 
School Budget 



Personality variables related to the 
mentally retarded. 



Initiate research, incentive classes 
for overaged seventh and eighth 
graders, evaluation of various dis- 
crict^A/ide projects . 

Improving student achievement for 
Hispanic-background children. 

Rehabilitating and educating severely 
disabled homebound persons. 



Reevaluation of a guidance program. 
Followup of high school graduates. 



Secondary school curriculum improve- 
ment study in mathematics. 

Evaluation of curriculum materials in 
grades K-12, 



Predicting academic success in 
elementary school. 



1) Project talent search. 2) Evalua- 
tion of student volunteer work with 
culturally deprived children, sex 
differences in scholastic achievement. 

Improving school attitudes and moti- 
vation by team- teaching and 
flexible grouping. 

Evaluation of special classes for 
emotionally disturbed, brain injured, 
learning problem children. 

Cognitive simplicity, school entrance 
age and achievement, evaluation of 
readiness for school, independent 
research. 
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Mr. D. F. Boyd 

IBM, Yorktown Heights 

IBM 

William S. Vincent 

Teachers College, Columbia Univ. 

School Systems 

Mr. Gerald S, Hauna 

Harcourt, Brace, and World, Inc. 

Harcourt, Brace, and World, Inc. 

Gary J. Robertson 

Harcourt, Brace, and World, Inc. 

Harcourt, Brace, and World, Inc. 



Mr. Thomas P. Hogan 

Harcourt, Brace, and World, Inc. 

Harcourt, Brace, and World, Inc. 



Simulation modeling of a school system. 



A variety of projects on quality 
measurement. 

Prediction of success in Algebra and 
Plane Geometry using aptitude tests 
and selected nontest variables. 

Test standardization and validation 
studies: Otis-Lennon Mental Ability 
Test and/or analysis of learning 
potential . 

Interpretation of reading difficulties 
and suggested remediation will be 
programed for a diagnostic reading 
test. 



New York State Education Department 
Office of Research and Evaluation 
Research Training Program 
March 1967 
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Appendix E 

NEW YORK STATE GRADUATE RESEARCH TRAINING PROGRAM 



SUMMARY OF INTERNSHIP EXPERIENCES 
1968 - 1971 



This report is based upon the questionnaires returned from 6 9 of the 77 
graduates of the program. The report includes the questionnaire 
with data summarized by year and by institution* Data are reported in 
means and percentages. 



The University of the State of New York 
The State Education Department 
Research Training Program 
Albany, New York 12224 
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Section I 



Analysis of Internship Questionnaire 
by Year of Completion 







Class of 


Class of 


Class of 


Class of 


Combined 




Quest ionnai re Items 


1968 


1969 


1970 


19~1 


data 






19 grade 


14 Rrado 


26 erad. 


10 Erad. 


69 erad. 


I. 


Number of days spent on 














internship 


120 


130 


120 


120 


122 




ao How many of these were 














£ ul 1 fiavs? 


110 


123 


108 


115 


112 




bo How many of these were 














half days? 


20 


14 








2« 


How many of these days were 














spent at 








74 


11 




ap A central office 


67 


100 


73 




b« In the field 


38 


Id 


4,0 




97 




Co Other locations 














(libraries, meetings. 














etCo ) 


20 


17 


19 


13 


18 


3, 


How many supervisory visits 














were made by the university 














director with you and the 










2.8 




director of your internship? 


2<.2 


lo2 


4.7 


1.5 


4. 


How many conferences did 














vnii Vi^^vp Abniit" the intern* 














ship with the university 








5.6 


6.3 




director? 


5,2 


3«9 


8.7 




Which statement best de- 














scribes the written 














reports required by the 














university for your intern- 














ship? 














a« No written reports 








56% 


43% 




requiredo 


47% 


28% 


46% 




bo One end-of-the- 














internship report 














describing in general 














the work of the 










23% 




internship 


11% 


36% 


35% 


0% 




c« Frequent detailed 














reports about the 














internship (cog.; 














monthly activity 








44% 


34% 




logue) 


42% 


36% 


19% 



ERIC 
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Class of 


Class or 


Class of 


Class of 


Combined 






Questionnaire Items 


1 n ^ o 
19 DO 


1969 


1970 


1971 


data 








19 erad. 


14 grade 


26 erad. 


10 grade 


69 grade 


0 « 


Were regular seminars or 














class meetings held at the 














university for all 














trainees in internship 














Positions? 


Yes 58% 


Yes 50% 


Yes 56% 


Yes 50% 


55% 








No 42/0 


No 5O/0 


No 44% 


No 50% 


45% 


7 


How many credits did you 














receive for the internship? 


o /. 


"7 0 

7c8 


8.9 


9<,3 


8. S 




During the internship, I 














worked on 














a. 


One major project 


16% 


36% 


31% 


10% 


25% 






exclusively 














b. 


2-3 projects 


32% 


21% 


11% 


20% 


20% 




c . 


4-5 projects 


D/o 


29% 


23% 


40% 


22% 




d. 


6 or more projects 


/ "707 


14% 


35% 


30% 


33% 




Rank the following 














activities by the amount 














of 


time you devoted to 














each; 














a. 


reading and library 
















research 


3c 1* 


3.6* 


4.9* 


4,2* 


3.9* 




b. 


meetings, planning. 
















sessions, and confer- 
















ences 


3o / 


0 "7 

3.7 


3*1 


3 c6 


3*4 




Co 


testing, interviewing. 
















and data collecting 


0 "7 
Z. / 




3c 3 


2oi 


3.1 




do 


developing data- 
















collecting instruments 




J • 0 




^« 3 






e* 


performing statistical 
















calculations 


5.9 


5.0 


5.1 


3.5 


5.0 




fc 


clerical tasks ( typing. 
















f ii ing, scoring) 


7.2 


6o4 


5.9 


6,7 


6.5 






writing proposals. 
















reports, and articles 




3.5 


4.0 


5.0 


4.0 




h. 


writing computer 
















programs 


7.4 


5.4 


7.6 


7.6 


7.5 




i * 


other (please list) 


7.1 


3.6 


7.3 


7.9 


7.6 



*These data are the average ranks. The lower the average the greater the 
amount of time the trainee devoted to the activity. 
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Ques t i onnai re It ems 



10. How frequently did you 
confer with your project 
director on your work? 
So almost daily 
b. once in a 5 -day week 
c* as often as I felt it 

necessary 
d as often as the 

director felt it 

necessary 
e* not frequently enough 

11* Were you responsible to 
persons other than the 
project director? 



12. Was the internship well 
equipped for research 
and evaluation tasks? 



13o Which of the following 
were readily available? 
a* typing service 

b. desk, telephone, 
work space 

Cu calculators 

d. library and reference 

materials 
e* computer services 
f. test files 

14o Did the internship 

necessitate your working 
with (equivalent of 2 
or more days)? 
a. teachers 

b« school administrators 

c. other school staff 
d* college faculty 

e* other researchers 
f» government personnel 
gc parents, boards of 
education members, 
communi ty 1 eader s , 
representatives of 
organizations 
ho children 
i» others 



Class of 


Class of 


Class of 


Class of 


Combined 


1968 


1969 


1970 


1971 


data 


19 Erad. 


14 £rad. 


26 Erad. 


10 erad. 


69 Rrad 


5% 


43% 


23% 


60% 


26% 


16% 


0% 


8% 


10% 


97o 


63% 


50% 


54% 


50% 


547o 


11% 


0% 


11% 


20% 


77o 


5% 


7% 


4% 


0% 


h% 


Yes 74% 


Yes 50% 


Yes 38% 


Yes 50% 


Yes 527o 


No 26% 


No 50% 


No 62% 


No 50% 


No 487o 


Yes 84% 


Yes 64% 


Yes 81% 


Yes 70% 


Yes 777o 


No 16% 


No 36% 


No 19% 


No 30% 


No 23% 


89% 


100% 


96% 


100% 


96% 


95% 
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Questionnaire Items 



15. Which 3tatement best 

describees your activities 
during the internship? 
ao The procedures for the 
project( s) were pre- 
established o I simply 
carry them out* 
bp Procedures for projects 
were outlined and 
flexible* I could 
suggest changes and 
when justified they 
were accepted. 
c« Few if any procedures 
were establishedo I 
developed and applied 
those appropriate for 
the project(s)o 

16o Which statement best 

describes your evaluation 

for the internship? 

ao An extremely valuable 

experience* All intsrns 

should have it. 
b. Valuable, but other 

internships should be 

considered before this 

one . 

Cft Of minimal value in the 
preparation of an edur 
cational researcher* 
If possible, no other 
interns should be 
placed here* 
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14 grad. 
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Appendix F 



Location of Students for Internships 



Intern 

(7/69-6/70) 
Neckers, Fred 



Rosensweig, Larry 



Roth, William 



Storte, John 



(9/70-6/71) 
Brogan, John 



Spadafora, Raymond 



The City University of New York 

Location 

Research Training Program 
N.Y.S. Education Department 
Albany, New York 



Supervisor 

Louis T. Di Lorenzo 
Thomas F. Gould 



City College David Fox 

The City University of New York 
New York, New York 

City College David Fox 

The City University of New York 
New York, New York 



City College 

The City University of Npw York 
New York, New York 



Theodore Abramson 



City College David Fox 

The City University of New York 
New York, New York 

City College David Fox 

The City University of New York 
New York, New York 



Intern 

(9/67-6/68) 
Abramson, Theodore 



Halpern, Shelley 
Harckham, Laura 
Katz, Alexander 



Fordham University 
Location 

S.U.T.E.C* 

Public School #76 

Long Island City, New York 

Center for Urban Education 
New York, Nev; York 

Ramapo Central School Dist. 
Spring Valley, New York 

Ferkauf Graduate School 
Yeshiva University 
New York, New York 



Supervisor 
Elaine B. Chapline 

Mortimer Kreuter 
Joel Elkind 
Julian Roberts 



ERLC 



-68- 



Intern 

(2/68-1/69) 
Polk, Virginia 



Spollen, Joseph 



Strum, Irene 



(9/68-6/69) 
Rivera, Luis 

(7/69-6/70) 
Flynn, Mary 



Oxman, Wendy 



Manchester, Harry 



(9/70-6/71) 
Butler, Sr. Loretta 



Location 

S.U.T.E.C. 

Public School #76 

Long Island City, New York 

Suffolk Educational Center 
Patchogue, New York 

Board of Cooperative Educa- 
tional Services 
Yorktown Heights, New York 



NoY.C. Board of Education 
Brooklyn, New York 



Nassau County Board of Coop- 
erative Educational Serv. 
Jericho, New York 

Institute for Research and 

Evaluation 
Fordham University 
New York, New York 

S.U.T.E.C. 

Public School #76 

Long Island City, New York 



Mineola Arts Project 
Mineola Public Schools 
Mineola, New York 



Supervisor 
Elaine B. Chapline 

Victor Gerhard, Jr. 
Richard L. Wing 

Phillip Bolger 
Jack Tanzman 
Joseph Justman 



Janet Brown 



Dennis Murphy 



Intern 

(2/68-1/69) 
Balletta, Robert 



Sanna, Margaret 



Simon, Alan 



Simon, Alan 



Hofstra University 

LoCoLion 

Connetquot Central Sch. Dist. 
Bohemia, New York 

New York Institute of 

Technology 
Old Westbury, New York 

The Responsive Environment 

Center 
Brooklyn, New York 

Division of Research 
N.Y.S. Education Department 
Albany, New York 



Supervisor 
John Cirincione 

Leonard Morton 
Benjamin Israel 
George Thomas 
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Intern 



Location 



Supervisor 



Wood, Edwin 



(9/68-6/69) 
Cohen, Edward 



Murphy, Dennis 



Woog, Pierre 



(7/69-6/70) 
Kay, Claire 



Schnide, Helen 



(9/70-6/71) 
Beban, Wallace 



Brown, Rita 



Mulcahey, Thomas 



Freeport Public Schools 
Freeport, New York 



The Education Council 
Mineola, New York 

Educational Development 

Laboratories , Inc . 
Huntington, New York 

Suffolk County Regional Center 
Patchogue, New York 



Freeport Public Schools 
Freeport, New York 

Union Free School Distri^ t 
Levittown, New York 



Nassau County Board of Co- 
operative Educational Serv. 
Jericho, New York 

Freeport Public Schools 
Freeport, New York 

Nassau Countv Board of Co- 
operative Educational Serv. 
Jericho, New York 



John Gordon 

Jack Tanzman 
Donald Senter 

John Keough 

AlQnzo Shockley 



Jerome Notkin 
Monroe Fremed 



Jack Tanzman 



Alonzo Shockley 



Jack Tanzman 



Intern 

(9/67-6/68) 
Shea, James 



(2/68-1/69) 
Carson, John 



New York University 

Location 

Long Beach Public Schools 
Long Beach, New York 



U.S.O.E., Region II 
New York, New York 



Supervisor 
Joseph Sturm 

John Sokol 



St. John's University 

Intern Location Supervisor 

(9/67-6/68) 

Gorman, Sr. M. Helen Manhattan College Frank Lodato 

Bronx, New York 
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Intern 

(2/68-1/69) 
Keller, Joan 

(7/69-6/70) 
Di Cesare, Vito 



Reilly, William 

Sullivan, Raymond 

Zygadlo, Henry 

(9/70^6/71) 
Goldberg, .n 



(1/71-7/71) 
Millar, Eric 



Location 

The Education Council 
Mineola, New York 



Supervisor 
William Callahan 



Suffolk County Regional Center John Keough 
Patchogue, New York 



Board of Cooperative Educa- 
tional Services 
Jericho, New York 

N.Y.C, Board of Education 
Brooklyn, New York 



William Callahan 



Philip Bolger 



Connetquot Central Sch. Dist. George Graham 
Bohemia, New York 



School District #13 
N.Y.C. Board of Education 
Brooklyn, New York 



Connetquot Central Sch. Dist. 
Bohemia, New York 



Louis T, Di Lorenzo 



Henry H. Zygadlo 



Intern 

(9/67-6/68) 
Hayden, Robert 



Locasclo, David 



Hofmann, Richard 



State University of New York at Albany 

Location 

Schenectady Public Schools 
Schenectady, New York 

Research Training Program 
N . Y,S . Education Department 
Albany, New York 

Department of Educational 

Psychology 
State Univ. of N.Y. at Albany 
Albany, New York 



Supervisor 
Clarence J. Spain 

Louis T. Di Lorenzo 
Leonard V. Gordon 



Murdoch, L. Robert 



Schenectady Public Schools 
Schenectady, New York 



Clarence J. Spain 



Intern 



O'Neal, Zenobia 



(2/68-1/69) 
Scott, Kathleen 

(9/68-6/69) 
Locascio, David 



Gould , Thomas 



(7/69-6/70) 
Byrne, Carolyn 



Kelliher, Paul 



Spath, Guy 



Pruner, James 



(9/70-6/71) 
Archer, Phillip 



I tzkowi t z , Michael 
Sewall, Michael 



Location 

School of Education 

State Univ. of N.Y. at Albany 

Albany, New York 



Albany City Public Schools 
Albany, New York 



Schenectady Public Schools 
Schenectady, New York 

Research Training Program 
N.Y.S. Education Department 
Albany, New York 



Utica Public Schools 
Utica, New York 

Office of Research and 

Evaluation 
N.Y.S. Education Department 
Albany, New York 

Research Training Program 
N.Y.S. Education Department 
Albany, New York 

School of Education 

State Univ. of N.Y. at Albany 

Albany, New York 



Office of Research and 

Evaluation 
N.Y.S. Education Department 
Albany, New York 

Schenectady Public Schools 
Schenectady, New York 

Office of Research and 

Evaluation 
N.Y.S. Eoucation Department 
Albany, New York 



Supervisor 
Richard Clark 



Conwell Higgins 



Clarence J. Spain 



Louis T. Di Lorenzo 



John H. Rosenbach 
Paul Baker 

Leo Doherty 



Louis T. Di Lorenzo 
Thomas F. Gould 



John H. Rosenbach 



Louis T. Di Lorenzo 
Robert O'Reilly 



Clarence J. Spain 



Louis T. Di Lorenzo 
Robert O'Reilly 
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State University College at Brockport 



Intern 

(6/69-7/70) 
Barry, G, Michael 



Messerich, Charles 



Miller, James 



Ogle, Robert 



Throop, Robert 



Titus, David 



Yoffredo, Ralph 



(9/70- 12/70) 
Rasmus sen, Peter 



(9/70-6/71) 
Brown, David 



Zillioux, M. Kathleen 



(9/71-6/72) 
Siebert, Robert 



Location 

Genesee Valley School Develop- 
ment Association 
Rochester, New York 

Kodak Research Center 
Rochester, New York 

Educational and Cultural 

Center 
Syracuse, Nev; York 

Genesee Valley School Develop- 
ment Association 
Rochester, New York 

Educational and Cultural 

Center 
Syracuse, New York 

Gates Chili Central Sch, Dist. 
Rochester, New York 

Campus School 

State Univ, Coll, at Brockport 
Brockport, New York 



Board of Education of the 

Virgin Islands 
Charlotte Amalie, St, Thomas 
Virgin Islands 

Research Training Program 
State Univ. Coll. at Brockport 
Brockport, New York 

Genesee Valley School Develop- 
ment Association 
Rochester, New York 



Supervisor 
Charles Walker 

Raymond Kicklighter 
Luton R. Reed 

Charles Walker 

Luton R. Reed 

Frank Denshaw 

Harry Emmerson 

Philip Gerard 

Louis T. Di Lorenzo 
Charles Walker 



Research Training Program Louis T. Di Lorenzo 

State Univ. Coll. at Brockport 
Brockport, New York 
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Intern 

Weaver, Francis 
Zusman, Richard 



Location 

Horseheads Public Schools 
Horseheads, New York 

Research Training Program 
State Univ. Coll. at Brockport 
Brockport, New York 



Supervisor 
Edward McHale 

Lou^s T. Di Lorenzo 



Intern 

(9/67-6/68) 
Harriger, James 



Rossi, Dominic 



(2/68-1/69) 
Pietropaolo, Joseph 



Regan, Frances 



Syracuse University 

Location 

Eastern Regional Institute for 

Education 
Syracuse, New York 

Eastern Regional Institute for 

Education 
Syracuse, New York 



Finger Lakes Regional Educa- 
tional Center 
Homer, New York 

Educational and Cultural Ctr. 
Syracuse, New York 



Supervisor 
Sidney M. Archer 

Sidney M. Archer 

Ernest Rookey 
Nicholas Collis 



Intern 

(9/67-6/68) 
Cinque, Carmela 



Barigliano, Leonard 

Gould, Thomas 
Szczypkowski, Ronald 



Teachers College, Columbia University 

Location 

Teachers College, Col. Univ. 
New York, New York 

Board of Cooperative Educa- 
tional Services 
Yorktown Heights, New York 



Supervisor 
Howard Fehr 

Benjamin Cohn 



Harcourc, Brace and World, Inc. Gary J. Robertson 
New York, New York. 



Bureau of Occupational and 

Vocational Research 
N.Y.S. Education Department 
Albany, New Y^^rk 



Louis Cohen 
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erJc 



Intern 

(2/68-1/69) 
Franklin, Ellen 



Gitlitz, Alfred 

McLaughlin, James 
Shaffer, Michael 



C7/&9-6/70) 
Ellis, Ronald 



Greene, Martin 



Scheer, Jeffrey 



Location 

Teachers College, Col, Univ. 
New York, New York 

Silver Burdett Division 
General Learning Corporation 
Morristown, New Jer.<;ey 

N.Y.C. Board of Education 
Brooklyn, New York 

Harcourt, Brace and World, Inc. 
New York, Mew York 



Research and Deronstration Ctr, 
Teachers College, Col. Univ. 
New York, New York 

Research Training Program 
N.Y.S. Education Department 
Albany, New York 

The Psychological Corporation 
New York, New York 



Intern 

(9/67-6/68) 
Andrews, Gloria 



Kenny, J. Ruport 



Raub, J. Robert 



Walker, Charles 



University of Rochester 

Location 

Genesee Valley School Develop- 
ment Association 
Rochester, New York 

Genesee Valley School Develop- 
ment Association 
Rochester, New York 

Genesee Valley School Develop- 
ment Association 
Rochester, New York 

Genesee Valley School Develop- 
ment Association 
Rochester, New York 



Supervisor 
Miriam Gold erg 

Ira Singleton 

Leonard W, Ingraham 
Harold Bligh 

Ross Evans 

Louis T. Di Lorenzo 
Thomas Gould 



Jerome Doppelt 
Robert North 

Supervisor 
Byron Williams 

Byron Williams 

Byron Williams 

Byron Williams 
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Appendix I 



Graduate Questionnaire on Employment and Attitudes 



DIVISION OF EVALUATION 
ALAN G. RODERTSON 
DIRECTOR 



BUREAU OF URBAN AND COMMUNITY 
PROGRAMS EVALUATION 
LEO D. DOHERTY, CHICF 
SID- 474.7277, 6323 



The Bureau of Urban and Community Programs Evaluation has been 
assigned the responsibility of developing a 5 year evaluation report 
to the U. S. Office of Education of the "Program for Training Educational 
Research Personnel for School Service." 

Since you were a participant in this Research Training, the Bureau 
would greatly appreciate your responses to the questionnaire enclosed. 
The few moments it will take to complete the questionnaire will provide 
this Bureau with valuable background and current data to De included in 
the report. Since the Bureau is under pressure to submit this report 
very soon, your prompt attention would be appreciated. 

If you have any questions regarding the oi ^ tionnaire, please do 
not hesitate to contact me or Mrs. Mary Horan 474-6325 or 474-3888* 

Sincerely, 



Leo D. Doherty 



The UnivorsiLy of the State of Ngw York 
THE STATE EDUCATION DEPARTMLNT 
Bureau of Urban and Community Progranis Evaluation 

Name:: Please return to: Mrs. Mary Horan 

State Education Department 
Address: Bureau of Urban & Community 

Programs Evaluation 
Institution at which you were a Washington Avenue 

participant in the program: Room 462 

Albany, New York 12224 

Dates: 

Program Director: 

1 . Emp lo yment History 

a. Description and location and approximate salary of employment before entrance into 
the program^ 



b. Description (major duties), location, and approximate salary of employment after 
completion of program* 



c. Description (major duties) and location of employment currently (if different 
frcn b). 



d* If engaged in evaluation and/or research activities, please give approxirnate 
percent of time spent on each duty, as listed in richer b and/or c. 



Employment expectations (what are your major employment goals) how do they differ 
from those when you entered the program? 
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Ac ad cm i c Status 

a. Highest degree held (date and name of institution awarding degree). 



b. If additional courses were taken after completion of the training program, please 
indicate name of course, credit hours, and name of institution where taken. 



c. Number and type of certificates (if any) you hold. 



Program / .titudes 

a. In view of your current employment, of what relav^-.nce was the training program' 

high explain 

medium explain 

low explain 



Are there any current research and/or evaluation problems with which you are 
dealing for which you feel the program did not fitdequatcly train you? 



c. Have your attitudes toward the program changed since you completed your training? 
(if so, in what respect) 
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d« What changos vould you recommend to st ronfilhen the kesearch Training Program? 



e. What was the most valuable part of the Research Training Program? 



f. Did you believe that the program would offer you more than you actually received? 



Publications 

A. Have you authored (individually or jointly) any articles on research and/or 
evaluation or in the genera^ fiela of education since you completed the program 
(please give appropriate bibliographical data)? 

1. 

2. 

3. 

4. 

B. What studies (supported by federal (e.g. Title I), state (e.g. Urban Ed.), local 
funds or others) have you conducted since completion of the program (please 
give appropriate bibliographical data)? 

1. 

2. 

3. 

4. 

C If you have made any presentations at conferences, please list with titles and 
dates and name of conference. 

1 . 

2* 

3. 

4. 
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ABSTRACT 

The "Griggs vs, Duke Power" decision of the Supreme 
Court specifies that job relatedness is the only lawful reason for 
using tests for the purpose of selection when their use results in 
the disproportionate exclusion of minority group members. One of the 
main implications of the "Griggs" decision for the test producer and 
user alike is a renewed emphasis on sound practices of employee 
selection* The producer of employment tests must put more effort into 
job analysis activities, criterion development activities, and 
criterion related validity studies. Another general concern for the 
test publisher is about certain characteristics of the test or its 
administration that are irrelevant to the specific abilities being 
measured but which depress scores for members of one group. Another 
consideration is that of differential validity studies. There is more 
than one reasonable definition of test fairness, and these 
definitions are in conflict; thus, competing values must be weighed. 
The test maker cannot resolve the problem of competing values, but he 
can provide the test user with information that will make explicit 
the value implications of various uses of tests. Two figures 
illustrate Thorndike's alternate definition of culture fairness. 
(CE) 
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Some Implications of the Griggs Decision 
for Test Makers and Users 



Robert L. Linn 

'■^^ Griggs vs . Duke Power decision the Supreme Court clearly identified 
job-re latedness as the only lawful reason for using tests for the purpose of 
selection when-^heir use results in the disproportionate exclusion of minority 
group members. In the unanimous decision delivered by Chief Justice Burger, 
he wrote: "If an employment practice which operates to exclude Negroes can- 
not be shown to be related to job performance, the practice is prohibited."""" 
Where tests are demonstrated to be job related, however, the Griggs decision 
makes it clear that tests are judged to be legal and useful". In the words of 
Chief Justice Burger, "Nothing in the [Civil Rights] 'Act [of I964, Title VII] 
precludes the use of testing or other measuring procedures; obviously they are 

useful What Congress has commanded is that any tests used must measure the 

person for the job and not the person in the abstract."^" 

The necessity of demonstrating job-relatedness is clear, but the evidence 
that is required for the demonstration is less clear. The EEOC Guidelines on 
Employment Selection Procedures^ and some of the litigation following the 
Griggs decision help identify the type of evidence that will probably be re- 
quired. The EEOC Guidelines are given specific support in the Griggs decision 
where it is argued that there is "...good reason to treat the Guidelines as 
expressing the will of Congress."^ Given this support, the Guidelines would 
seem to be the natural place for the employer, the test producer, and the 
personnel psychologist to turn in order to insure that the use of tests for 
employee selection or promotion is consistent with the requirements of the 
Civil Rights Act of I964. 
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The G.uidolineg stress the importance of careful job analysis. The 
job analysis provides the basis for test selection. It also may provide 
the basis for defending the rational validity of the test or, preferably, 
the development of appropriate criterion measures to be used in the empirical 
investigation of ^criterion-related validity. Bill Enneis has pointed out 
that the Guidelines- "...embody the substance f good personnel employment 
practices as recommended by experts in industrial psychology and personnel 
adminisT.ration for the past forty or fifty ysars."^ I'm in general agree- 
ment with this evaluation. The reader of the Guidelines can certainly find 
much in common with professionally accepted prescriptions for personnel 
selection. While generally agreeing with this position, Guion has identified 
two aspects of the Guidelines that seem to be new but not "...especially 
heretical to orthodox testers."^ The two new features are (l) an expanded 
definition of tests and (2) the emphasis on independent validation for minority 
groups (i.e., the emphasis on differential validity studies). I'll have more 
to say about the emphasis on differential validity a bit later. 

From the above perspective, one of the main mplications of the Grigg s 
decision for the test producer and the user alike is a renewed emphasis on 
sound practices of employee selection. It is no longer sufficient to use a 
test because it reliably measures general ability. It must be shown that 
the test is measuring an attribute that is iraportant for successful job 
per fo nuance. In some cases professionally aeveloped tests of performance 
(e.g. , a tyoing test) may stand on the rational relationship of the perform- 
ance to the analysis of the job. For more general ability tests, however, 
criterion-related validity or a strong combination of construct validity 
coupled with an analysis of the mportance of the constructs to job perform- 
ance will probably be required. 
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To provide the necessary backup for test users the producer of 
employment tests is going to have to put more effort into jc'b analysis 
activities, criterion development activities, and criterion related validity 
studies than many of them have done in the past. For many, if not most 
jobs, however, the magnitude of th^ -effort that is required is tremendous. 
Judging from the ETS--Civil Service Commission study of three occupations 

n 

that was directed by Joel Campbell,^ it seems unreasonable to expect that 
the typical local situation will permit the kind of comprehensive study 
that might be desired. Careful attention to comprehensive job analyses, 
which were used to develop objective criterion measures and select tests 
that measured abilities that were judged to be "unportant for good performance 
on the job, paid off in impressive validities in Campbell's stud/. However, 
that study required several years to accomplish as well as substantial 
support from the Ford Foundation, the close cooperation of the Civil Service 
Commission, and contributions from a ni:anber of LTS and Civil Service staff 
members. Reflection on the magnitude of the effort in Campbell's study in 
contrast to the resources that are typically available in the real-life per- 
sonnel situation led Anastasi to ask: 'Tn a more nearly typical personnel 

situation, what... can be done to ensure that selection tests are truly valid, 

g 

or relevant to the job?" In response to her own question Anastasi answered: 
•'For this purpose... I would turn to a thorough, professional job analysis, 
followed by a study of the published findings regarding the validity of 
different tests against specific job functions."^ She urged ^'... that more 
effort be expended on basic research regarding the specific aspects of be- 
havior measured by different instruments and less on inadequate, inconclusive, 
local validation studies against global criteria of job performance."'''^ I 



would heartily endorse this pofjition. It seems to me to be very consistent 
with both the need for better construct validity and Law.she^s notion of 
synthetic validity ."^"^ It also suggests an important role for the test 
producer, namely, that of providing the necessary research base required 
to understand the aspects of behavior that are measured by their instru- 
ments so that the personnel psychologist will have a soimd basis for using 
his job analysis to select tests. 

While obviously stimulated by the need to ensure that employment 
tests are used appropriately with minority groups, the above concerns 
and recommendations are actually quite general and should apply to all 
tesT^ing — not just testing of minority group members* Another general 
concern for the test publisher that is given spJliefx^.^^ by questions of 
equaJ. emplo^nnent opportunities for minority group members is the concern 
about certain characteristics of the test or its administration that are 
irrelevant to the specific abilities being measured but which depress 
scores for members of one group. The classic example is the test that 
requires considerable verbal ability or has difficult reading requirements 
when these are not the skills that are being measured. The context of the 
test items or specific words may be more familiar to one group than another, 
and where these factors are not essential to the attribute being measure'-l, 
they should be eliminated. One way to ensure this is for the test producers 
to obtain the input from representatives of the various groups who will 
be taking the test. In other words, minority group members need to be 
represented in all phases of the test construction process (e.g. , test 
specifications, item writing, and reviews of the test). Comparative item 
analyses for different groups of people are also potentially useful for 
this purpose. 
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Another consideration is that of differential validity studies. As 

was mentioned earlier, the iilZOC Guidelines stress the importance of such 

studies. "Data must be generated and results separately reported for minority 

12 

and nonminority groups wherever technically possible." I would r_ot argue 
against the desirability of differential validity studies where feasible; 
however, it may be worth^;hile to consider the implication of evidence that 
is accumulating from the differential validity studies that have been 
conducted to date. Obviously the evidence is not complete, but a fairly 

substantial number of studies have been conducted in the past few years. 

% 

At the recent meetings of the American Psychological Association, 

William Ruch reviewed differential validity studies that v/ere conducted ir 

a nonmilitary business or industrial situation and that had separate 

statistics reported for blacks and whites which permitted him to test for 

homogeneity of regression for the two groups. He also required that race 

not be confounded with some other variable (e.g., blacks working for one 

company and whites for another). From his analyses, Ruch concluded: 

"Certainly these 20 studies do not tell the whole story. The evidence they 

do provide is that there is no such thing as differential validity but 

13 

there is a tendency of tests to overestimate black job performance.'' 

Ruch is by no means the only person to conclude that tests which 
are valid for one group are usually valid for the other (i.e., differential 
validity is rare) and that where differences in the prediction systems 
are f'^und, the predictions based on the total group usually tend to over- 
estimate rather than underestimate minority group performance on the 
criterion. For example, this conclusion is reached by Bray and Moses in 
their chapter in the 1972 Annual Revjew of Psychology . and is one of 
the clear results of Campbell's Civil Sei^ice Study*^^ v/hich I mentioned 

ERIC 
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previously. In a military setting, a similar conclusion is reported by 

16 

Guinn, Tupes and Alley, and in an educational setting the tendency is 

17 

also found to be in the direction of overprediction. 

Inasmuch as the overprediction finding is valid,, it has ir.iplications 
tliat could run counter to one of the intents of the Civil Rights Act and 
to the reason for stressing differential ''*alidity studies in the Guidelines . 
If cutoff scores are ",..set so as to predict the same probability of 
success in both groups"*^^ as the Guidelines suggest, and the majority group 
regression line overpredicts for the minority group, the cutoff score v/ould 
be set higher for the minority f;roup than for the majority group. Thus, 
the differential validity concept which was intended to have a favorable 
influence on the employment of minority group members could be used to 
defend a policy that actually excluded more minority group members than 
would have been the case previously. 

There are many reasons \/hy the above use of such differential validity 
study results should be emphatically rejected. WeroS and I have argued else- 
where that there are statistical and psychometric reasons that could lead 

19 

one to expect the overprediction result. These are: (l) the failure to 
include a variable in the prediction system on which the groups differ and 
(2) the lack of perfect test reliability. There are also less subtle 
artifacts that could cause the overprediction result, for example, the criterion 
variable itself may not be free of bias. This is a particularly plausible 
explanation when supervisor ratings are used as the criterion variable. 

In nry vi^/w, however, there is a still more important reason for 
rejecting the possible implication of differential validity results that 
higher cutoff scores be required of the minority group than the majority 



group. This reason is that the maximization of performance on the job is 

only one value that needs to be attended to, and perhaps not the most important 

one . 

The usual definition of test fairness in terms of equal regression 
equations is fair to the institution in the sense that it provides the 
institution with a way of selecting people such that the average criterion 
score of those selected is a maximm. It is also "fair" to individual 
members of the two groups in that the criterion performance is not system- 
atically over- or underpredicted for members of either group. Nonetheless, 
Thorndike clearly demonstrated that a test which has equal regression 
equations for the two groups "...is ^unfair' to the lower scoring group 
as a whole in the sense that the proportion qualified on the test will 

be smaller, relative to the higher-scoring group, "than the proportion that 

20 

will reach any specified level of criterion perf oriiance . " 

Thorndike has argued persuasively that the traditional way of looking 
at test fairness in terms of equality of the prediction systems is not the 
only perspective that should be considered. The problem with the traditional 
definition that Thorndike has so clearly identified is illustrated in Figure .1. 
In Figure 1 a situation is depicted in .hich the regression of the criterion 
on the test is ^identical in the two groups (shown by the solid line with a 
slope equal to .25)* Thus, the test would be considered "fair" from the 
traditional psychometric point of view and according to the Guidelines . 
Note, however, that the difference in means on the test is substantially 
larger than the difference on the criterion variable. 



Insert Figure 1 about here 
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Now suppose for purposes of illustration thai: the mean of group A on 
the criterion variable corresponded to what was considered minimum acceptable 
performance (i.e., those with criterion scoies ebove that point are considered 
successes and those below it are considered failures). As can be seen in 
Figure 1, about ^0% of the hypothetical group A have criterion scores above 
this cutting point and about 20% of group B (shaded area of the group B 
distribution curve) have criterion scores above the cutting point. Since 
Criterion scores are not known in advance, it is the predicted criterion 
scores that are used for selection. If only those individuals with predicted 
scores equal to the success-failure point on the criterion variable were 
accepted, then approximately 30% of group A would be accepted but essentially 
none of the group B members would be accepted. This is the phenomenon that 
Thorndike was referring to when he said that while the traditional approach 
with equal regression equations is '*fair" to individual members of the minor 
group, it is "unfair" to the minor group as a whole, "...in the sense that 
the proportion qualified on the test, relative to the higher scoring grouD, 
will be smaller than the proportion that will reach any specified level of 
criterion performance." In the illustration depicted in Figure l,.tbe 
relative proportions in groups A and B that are qualified on the test are 
•50 and essentially zero, respectively, whereas the proportions above the 
success-failure point on ::he criterion variable are approximately ,50 and 
.20 respectively. 

Thorndike proposes that: "An alternate definition [of fairness] would 

Specify that the qualif^dng scores on the tests should be set at levels 

that will qualify applicants in the two groups in proportion to the 

fraction of the two groups reaching a specified level of criterion per- 
22 

Q formance." Thus, we have two conflicting definitions of fairness. The 

ERIC 
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only time that the equal regression definition and Thorndike's definition 
are in agref^ment is when the validity is perfect. This unrealistic case 
is depicted by the 45^ line in Figure 2. With perfect validity and the same 
regression equation in both groups, the relative proportions qualified on 
the test (i.e., above the accept-reject point in Figure 2) are equal to 
the relative proportions that are successful (i.e., above the success- 
failure point in Figure 2). 

Insert Figure 2 about here 

The requirement that an equal proportion of each group be accepted 
as would be successful also could be satisfied by a situation where the 
regression line for the group B lies below the one for group A (i.e., 
the group A equation overpredicts for group B) but the cutting score 
appropriate for group A is used to select within both groups. This 
situation is also depicted in Figure 2 by the two parallel lines with 
slopes of .25* 

In suinmary, Thorndike has identified a flaw in the traditional 
definition of ''fairness," His formulation suggests that we should be 
looking at the l..plications for the proportions of students admitted as 
well as the regression linevS. For a case with the same regression equations 
as that illustrated in Figure 1, different cutting scores would be required 
to make the proportions qualified on the test equal to the proportions 
successful on the criterion. In particular, a lower cutting score on the 
test would be required for the hypothetical group B in Figure 1 in order -^o 
satisfy Thorndike ^s alternate definition. 
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The probx^.T. that Thorndike identified is implicit in some earlier 
discussions ♦ For example. Cooper and Sobel seem to be referring to Lhis 
problem in their aiscussion of tests with lev/, but significant, validity. 
They argue that where such tests have an adverse impact on blacks, us 
of test scores as a major factor in employment decisions is likely to be 
unnecessarily prejudicial to blacks/' More tha:* one sense of fairness 
seems implicit in Cooper and Sobel ^s comment. 

In nry opinion, Thorndike 's main contribution is that of jraking it 
explicit that there is more than one reasonable definition of test fairne 
and that these definitions are in conflict. Thus, v/e must look beyond a 
simple technical resolution of the problem. Competing values must be 
weighed. Errors of selecting individuals v:ho are unsuccessful ana errors 
of rejecting applicants who would be successful must be v/eighed for 
members of minority groups and for members of the majority ^^oup. The 
former error is of prime concern in some employee selection situations 
where mistakes on the job can be very costly or even disastrous (e.g., 
the selection of pilots). In many situations, however, the latter type 
of error may be of greater concern. The test maker cannot resolve the 
problem of competing values, but he can provide the test user with in- 
formation that will make explicit the value implications of various uses 
of tests. 
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Figure 2 

Illustration of Thorndike's Alternate Definition 
of Culture Fairness 
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